PHASE 2 - QUALITY ASSESSMENT
WHY UNDERTAKE A QUALITY ASSESSMENT PROCESS?
The quality assessment process will provide for a more in-depth consideration of Australia’s essential statistical assets in order to understand how well the currently produced information meets the critical needs of users, highlighting gaps within existing statistics.
HOW WILL THE ASSESSMENT BE UNDERTAKEN?
The quality assessment will consider each essential statistical asset, including the datasets used in its production. This collective list of required datasets will facilitate the identification of duplication of statistical production across government as well as potential areas of underutilisation. Part of the process will involve reviewing the ability of the existing datasets against the critical spatial, frequency and other disaggregations identified in the 2013 list of ESA for Australia that each statistic is to be reported against.
The quality assessment phase of the ESA for Australia initiative will first undergo a pilot program, where a small number of essential statistical assets will be reviewed. The statistics chosen for the pilot will include ABS and non-ABS datasets at both the Australian Government, and state and territory level. The pilot will also be used to ensure that the process maximises the use of existing information where possible, and does not impose significant burden on data custodians.
A quality assessment tool leveraging off the existing ABS Data Quality Framework is being developed to assess each essential statistical asset against each of the seven dimensions of quality: institutional environment, relevance, timeliness, accuracy, coherence, interpretability and accessibility. The data quality framework is already used by a number of government agencies in the preparation of data quality statements to inform users on the quality of their statistics. Where possible, existing data quality statements will be used to support the quality assessment process in order to reduce the effort required for this phase.
This tool will use a series of questions designed to be easily answered with limited imposition on data custodians, with support from the ABS. The questions are developed to test how the available datasets match the needs highlighted through the ESA for Australia consultation process.
Where relevant, it will also gather information on what standards or classifications are being used, in order to test the comparability and coherence across the data sources. Based on the availability or otherwise of the critical information, the data source will be given a numerical score for each of the quality dimensions.
WHO WILL BE UNDERTAKING THE ASSESSMENT?
In its capacity as the national statistics office, the ABS will lead the quality assessment process. When a statistical asset involves a producer and/or data custodian other than the ABS, the quality assessment will be conducted in collaboration between the data custodian and the ABS to produce an agreed quality assessment.
HOW WILL THE RESULTS BE PRESENTED?
The purpose of generating a numerical score against each of the quality dimensions is to enable the results to be presented in a graphical format. The graphical interface will distribute the quality dimensions across separate axes and provide a benchmark across time.
This approach to quantifying data quality is consistent with current emerging trends internationally, such as the Food and Agriculture Organisation (FAO) within the United Nations (footnote 4). Another purpose of the pilot approach to this phase will be to test the approach to quantifying quality, including the application of weightings to specific questions and quality dimensions, and different approaches to presenting the results.
A quantifiable approach will also allow groups of statistics to be compared easily in a visual format across similar axes. This will help observe which quality dimensions the current statistics score highly on, and which ones can be improved. This ability to visualise quality will assist decision makers in assessing which elements of quality require improvement.
Given that the purpose of the exercise is primarily to identify areas for improvement across the range of essential statistical assets, it is not intended that detailed results for individual datasets will be made publicly available as a matter of course. The information will be shared with data custodians, and not circulated more widely without custodians’ consent. However, summary quality information for each statistic will be provided publicly.
RECOGNISING HIGH QUALITY INDIVIDUAL STATISTICS AND DATASETS
Once the quality assessment phase has been completed, in addition to the information being used for the purposes of the ESA for Australia initiative, there may be merit in seeking to publicly recognise statistics that meet the quality standards to a certain level.
One potential future approach to be discussed with data custodians is the consideration of an accreditation process, similar to the National Statistics Accreditation currently employed in the UK. This will not be the focus of Phase 2 but will be discussed with stakeholders as a potential future direction to gauge the level of support in future iterations of the ESA for Australia initiative.
MAKING THE QUALITY ASSESSMENT TOOL WIDELY AVAILABLE
After the quality assessment tool has been thoroughly tested and validated through the ESA for Australia process, it is possible that the ABS will seek to make the tool available for users and data custodians to undertake their own quality assessments for broader application to statistics not contained on the list of ESA for Australia, including non-official statistics.
This would be as a complementary tool to the currently available Data Quality Online Tool, available on the
NSS website which is designed to assist users in generating data quality statements. In the event that an accreditation process is eventually established, there is also the potential for accreditation to be extended beyond the statistics identified on the list of ESA for Australia.
Footnote
4. FAO Statistical Data Quality Framework: A multi-layered approach to monitoring and assessment, available at
http://unstats.un.org <Back
Print Page
Print All